feat(stt): conversation-aware recognition — keyterms + chat context#1967
Merged
Conversation
Port of livekit/agents#6039 (Python). Adds framework-managed STT keyterms with optional LLM-based auto-detection (KeytermDetector), plus a chat context sink so STTs with native carryover can bias recognition on the live conversation. - stt: `keyterms`/`chatContext` capabilities and `_updateSessionKeyterms` / `_pushConversationItem` hooks (forwarded by Fallback/Stream adapters) - voice: KeytermDetector with pending->confirm state machine, TTL eviction, turn-interval gating and forced record_keyterms tool call - plugins: deepgram v1/v2 + inference STT keyterm support (deferred while speaking), assemblyai keyterms + agent_context carryover - inference: strip `$schema` from tool JSON schemas — the zod v3 draft 2019-09 URI makes some gateway deployments end the stream with no tool call (pydantic never emits it, so this also matches Python) Co-authored-by: Cursor <cursoragent@cursor.com>
🦋 Changeset detectedLatest commit: bd34369 The changes in this PR will be included in the next version bump. This PR includes changesets to release 35 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Co-authored-by: Cursor <cursoragent@cursor.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…terms on bind - thread the detect task's abort signal into runOnce/detectKeyterms so aclose() unblocks an in-flight LLM pass immediately (parity with Python's asyncio cancellation) instead of waiting out the detection timeout; a cancelled pass no longer touches keyterm state - push the current (possibly empty) keyterm set to a keyterm-capable STT on first bind, so a reused STT instance can't retain session keyterms from a previous detector Co-authored-by: Cursor <cursoragent@cursor.com>
updateOptions was baking the active session keyterms into this.opts.modelOptions, so a later _updateSessionKeyterms pass read the polluted baseline and could never remove previously applied terms. Session keyterms are now only merged into what is sent to live streams and computed on the fly at connect time, mirroring the deepgram and assemblyai plugins' separate user/session tracking. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Port of the Python PR livekit/agents#6039 — conversation-aware STT recognition.
stt/stt.ts): newkeyterms/chatContextcapabilities and internal_updateSessionKeyterms/_pushConversationItemhooks (warn-once when unsupported).FallbackAdaptermerges capabilities and fans out both hooks;StreamAdapterdelegates to the wrapped STT.KeytermDetector(voice/keyterm_detection.ts): maintains static + auto-detected keyterms. On each user turn (gated byturnInterval, single-flight, 10s timeout) it snapshots the transcript and runs a forcedrecord_keytermstool call against a detection LLM (defaultgoogle/gemma-4-31b-itvia the inference gateway). Terms move pending → confirmed with TTL eviction and an optionalmaxKeytermscap; LLM metrics are re-emitted through the session.AgentSessionacceptskeytermsOptions, exposessession.keyterms, and supports updating static terms viaupdateOptions;AgentActivitystarts/stops the detector with the activity's STT and subscribesconversation_item_added→_pushConversationItemfor chat-context-capable STTs.keyterm), Deepgram v2/Flux (keyterms), and inference STT (provider-mappedextra) apply merged user + session keyterms, deferring mid-speech updates until end of speech. AssemblyAI adds keyterms plusagent_contextcarryover (agentContextCarryover, U3-Pro models) implementing thechatContextcapability.inference/llm.ts): strip$schemafrom tool JSON schemas. The zod v3 conversion embeds a draft 2019-09$schemaURI that makes some gateway deployments silently end the stream with no tool call; pydantic never emits the key, so this also matches what Python sends. (Non-mutating: destructured, so caller-supplied raw schemas are untouched.)Python's Google STT hunk is intentionally not ported (no Google STT plugin exists in JS).
Test plan
tests/test_stt_context.py(keyterm_detection.test.ts) — state machine, STT binding, chat context sink, triggering, helpers.agent_context.api:checkfailure is pre-existing onmain).